DACIDR: Deterministic Annealed Clustering with Interpolative Dimension Reduction using Large Collection of 16S rRNA Sequences

The development of next-generation sequencing technology has made it possible to generate millions of sequences from environmental samples. However, the difficulty associated with taxonomy-independent analysis increases as the sequence size expands. Most of the existing algorithms, which aim to generate operational taxonomic units (OTUs), require quadratic space and time complexity that makes them only suitable to small datasets. An alternative is to use heuristic methods; although it enables fast sequence analyzing, the hard-cutoff similarity threshold set for it and the random starting seed can result in reduced accuracy and overestimation. In this paper, we propose DACIDR: a parallel sequence clustering and visualization pipeline, which can address the overestimation problem along with space and time complexity issues as well as giving robust result. The pipeline starts with a parallel pairwise sequence alignment analysis followed by a deterministic annealing method of clustering and dimension reduction. No explicit similarity threshold is needed with the process of clustering. Experiments with our system also proved the quadratic time and space complexity issue could be solved with a novel heuristic method called Sample Sequence Partition Tree (SSP-Tree), which allowed us to interpolate millions of sequences with sub-quadratic time and linear space requirement. Furthermore, SSP-Tree can enhance the speed of fine-tuning on the existing result, which made it possible to recursive clustering to achieve accurate local results. Our experiments showed that DACIDR produced a more reliable result than two popular greedy heuristic clustering methods: UCLUST and CD-HIT.